Control method and apparatus, and electronic device

ABSTRACT

An image processing method and apparatus, and an electronic device are provided. The image processing method is applied in an electronic device. The electronic device includes an image sensor. The image sensor includes an array of photosensitive pixel units and an array of filter units arranged on the array of photosensitive pixel units, each filter unit corresponds to one photosensitive pixel unit, and each photosensitive pixel unit includes a plurality of photosensitive pixels. The image processing method includes outputting a merged image by the image sensor; determining a focusing area of the merged image; determining whether there is a target object in the focusing area; and when there is the target object in the focusing area, converting the merged image into a merged true-color image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation Application of U.S. application Ser.No. 15/812,401, filed Nov. 14, 2017, which is based on and claimspriority of Chinese Patent Application No. 201611079892.4, filed on Nov.29, 2016, the entire contents of which are incorporated herein byreference.

FIELD

The present disclosure relates to the imaging technology field, and moreparticularly to an image processing method, an image processingapparatus and an electronic device.

BACKGROUND

When an image is processed using a conventional image processing method,either the obtained image has a low resolution, or it takes a long timeand too much resource to obtain an image with high resolution, both ofwhich are inconvenient for users.

DISCLOSURE

The present disclosure aims to solve at least one of existing problemsin the related art to at least some extent. Accordingly, the presentdisclosure provides an image processing method, an image processingapparatus and an electronic device.

Embodiments of the present disclosure provide an image processingmethod. The image processing method is applied in an electronic device.The electronic device includes an image sensor. The image sensorincludes an array of photosensitive pixel units and an array of filterunits arranged on the array of photosensitive pixel units. Each filterunit corresponds to one photosensitive pixel unit, and eachphotosensitive pixel unit includes a plurality of photosensitive pixels.The image processing method includes: outputting a merged image by theimage sensor, in which, the merged image includes an array ofcombination pixels, and the plurality of photosensitive pixels in a samephotosensitive pixel unit are collectively output as one merged pixel;determining a focusing area of the merged image; determining whetherthere is a target object in the focusing area; when there is the targetobject in the focusing area, converting the merged image into a mergedtrue-color image.

Embodiments of the present disclosure further provide an imageprocessing apparatus. The image processing apparatus is applied in anelectronic device. The electronic device includes an image sensor. Theimage sensor includes an array of photosensitive pixel units and anarray of filter units arranged on the array of photosensitive pixelunits. Each filter unit corresponds to one photosensitive pixel unit,and each photosensitive pixel unit includes a plurality ofphotosensitive pixels. The image processing apparatus includes anon-transitory computer-readable medium including computer-readableinstructions stored thereon, and an instruction execution system whichis configured by the instructions to implement at least one of a firstcontrol module, a first determining module, a second determining module,and a first converting module. The first control module is configured tooutput a merged image by the image sensor, in which, the merged imageincludes an array of merged pixels, and the plurality of photosensitivepixels in a same photosensitive pixel unit are collectively output asone merged pixel. The first determining module is configured todetermine a focusing area of the merged image. The second determiningmodule is configured to determine whether there is a target object inthe focusing area. The first converting module is configured to convertthe merged image into a merged true-color image when there is the targetobject in the focusing area.

Embodiments of the present disclosure provide an electronic device. Theelectronic device includes a housing, a processor, a memory, a circuitboard, a power supply circuit and an imaging apparatus. The circuitboard is enclosed by the housing. The processor and the memory arepositioned on the circuit board. The power supply circuit is configuredto provide power for respective circuits or components of the electronicdevice. The imaging apparatus includes an image sensor. The image sensorincludes an array of photosensitive pixel units and an array of filterunits arranged on the array of photosensitive pixel units. Each filterunit corresponds to one photosensitive pixel unit, and eachphotosensitive pixel unit includes a plurality of photosensitive pixels.The memory is configured to store executable program codes. Theprocessor is configured to run a program corresponding to the executableprogram codes by reading the executable program codes stored in thememory, to perform the image processing method according to embodimentsof the present disclosure.

Additional aspects and advantages of embodiments of present disclosurewill be given in part in the following descriptions, become apparent inpart from the following descriptions, or be learned from the practice ofthe embodiments of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other aspects and advantages of embodiments of the presentdisclosure will become apparent and more readily appreciated from thefollowing descriptions made with reference to the drawings.

FIG. 1 is a flow chart of an image processing method according to anembodiment of the present disclosure.

FIG. 2 is a block diagram of an image sensor according to an embodimentof the present disclosure.

FIG. 3 is a schematic diagram of an image sensor according to anembodiment of the present disclosure.

FIG. 4 is a flow chart illustrating a process of determining a focusingarea of the color-block image according to an embodiment of the presentdisclosure.

FIG. 5 is a flow chart of an image processing method according toanother embodiment of the present disclosure.

FIG. 6 is a flow chart of an image processing method according to yetanother embodiment of the present disclosure.

FIG. 7 is a flow chart illustrating a process of converting acolor-block image into a simulation image according to an embodiment ofthe present disclosure.

FIG. 8 is a schematic diagram illustrating a circuit of an image sensoraccording to an embodiment of the present disclosure.

FIG. 9 is a schematic diagram of an array of filter units according toan embodiment of the present disclosure.

FIG. 10 is a schematic diagram of a merged image according to anembodiment of the present disclosure.

FIG. 11 is a schematic diagram of a color-block image according to anembodiment of the present disclosure.

FIG. 12 is a schematic diagram illustrating a process of converting acolor-block image into a simulation image according to an embodiment ofthe present disclosure.

FIG. 13 is a flow chart illustrating a process of converting acolor-block image into a simulation image according to anotherembodiment of the present disclosure.

FIG. 14 is a flow chart illustrating a process of converting acolor-block image into a simulation image according to anotherembodiment of the present disclosure.

FIG. 15 is a schematic diagram showing an image pixel unit of acolor-block image according to an embodiment of the present disclosure.

FIG. 16 is a flow chart illustrating a process of converting acolor-block image into a simulation image according to anotherembodiment of the present disclosure.

FIG. 17 is a flow chart of illustrating a process of converting a mergedimage into a merged true-color image according to an embodiment of thepresent disclosure.

FIG. 18 is a block diagram of an image processing apparatus according toan embodiment of the present disclosure.

FIG. 19 is a block diagram of a first determining module according to anembodiment of the present disclosure.

FIG. 20 is a block diagram of an image processing apparatus according toanother embodiment of the present disclosure.

FIG. 21 is a block diagram of a second converting module according to anembodiment of the present disclosure.

FIG. 22 is a block diagram of a third determining unit in the secondconverting module according to an embodiment of the present disclosure.

FIG. 23 is a block diagram of a second converting module according toanother embodiment of the present disclosure.

FIG. 24 is a block diagram of a second converting module according toanother embodiment of the present disclosure.

FIG. 25 is a block diagram of a first converting module according to anembodiment of the present disclosure.

FIG. 26 is a block diagram of an electronic device 1000 according to anembodiment of the present disclosure.

EMBODIMENTS OF THE PRESENT DISCLOSURE

Reference will now be made in detail to exemplary embodiments, examplesof which are illustrated in the accompanying drawings, in which the sameor similar reference numbers throughout the drawings represent the sameor similar elements or elements having same or similar functions.Embodiments described below with reference to drawings are merelyexemplary and used for explaining the present disclosure, and should notbe understood as limitation to the present disclosure.

In the related art, an image sensor includes an array of photosensitivepixel units and an array of filter units arranged on the array ofphotosensitive pixel unit. Each filter unit corresponds to and coversone photosensitive pixel unit, and each photosensitive pixel unitincludes a plurality of photosensitive pixels. When working, the imagesensor is controlled to output a merged image, which can be convertedinto a merged true-color image by an image processing method and saved.The merged image includes an array of merged pixels, and the pluralityof photosensitive pixels in a same photosensitive pixel unit arecollectively outputted as one merged pixel. Thus, a signal-to-noiseratio of the merge image is increased. However, a resolution of themerged image is reduced.

Certainly, the image sensor can be controlled to output a high pixelcolor-block image, which includes an array of original pixels, and eachphotosensitive pixel corresponds to one original pixel. However, since aplurality of original pixels corresponding to a same filter unit havethe same color, the resolution of the color-block image still cannot beincreased. Thus, the high pixel color-block image needs to be convertedinto a high pixel simulation image by an interpolation algorithm, inwhich the simulation image includes a Bayer array of simulation pixels.Then, the simulation image can be converted into a simulation true-colorimage by an image processing method and saved. However, theinterpolation algorithm consumes resource and time, and the simulationtrue-color image is not required in all scenes.

Thus, embodiments of the present disclosure provide a novel imageprocessing method.

Referring to FIG. 1, an image processing method is illustrated. Theimage processing method is applied in an electronic device. Theelectronic device includes an imaging apparatus including an imagesensor. As illustrated in FIGS. 2 and 3, the image sensor 10 includes anarray 12 of photosensitive pixel units and an array 14 of filter unitsarranged on the array 12 of photosensitive pixel units. Each filter unit14 a corresponds to one photosensitive pixel unit 12 a. In at least oneembodiment, there is a one-to-one correspondence between the filterunits and the photosensitive pixel units. Each photosensitive pixel unit12 a includes a plurality of photosensitive pixels 122. The imageprocessing method includes the following.

At block 211, the image sensor outputs a merged image.

The merged image includes an array of merged pixels. The plurality ofphotosensitive pixels in a same photosensitive pixel unit arecollectively output as one merged pixel.

At block 212, a focusing area of the merged image is determined.

At block 214, it is determined whether there is a target object in thefocusing area, if yes, an act at block 216 is executed.

In some embodiments, the target object includes a human face. Generally,there is not much requirement on the definition of an image including ahuman face, instead, the user may wish to realize a diffusion effect onthe face image, such that when the human face is detected in thefocusing area, it is only needed to directly convert the merged imageinto the merged true-color image without improving the definition of theimage, thereby avoiding a situation that flaws of the human face arealso clearly illustrated in the obtained image.

At block 216, the merged image is converted into a merged true-colorimage.

With the image processing method according to embodiments of the presentdisclosure, the image sensor can be controlled to output a suitableimage by identifying and determining the image in depth of focus, asituation that it takes too much work to output a high quality image bythe image sensor can be avoided, thus reducing work time of theelectronic device, improving work efficiency and improving the userexperience.

In some embodiments, after the merged image is output, it is determinedwhether a brightness of the merged image is less than a predeterminedbrightness threshold, if yes, it indicates that there is not enoughlight for this merged image, such that the merged image is convertedinto the merged true-color image to ensure the SNR (signal noise ratio).

Referring to FIG. 4, in some implementations, the act at block 212includes the following.

At block 2121, the merged image is divided into a plurality of analysisregions arranged in an array.

At block 2122, a phase difference of each of the plurality of analysisregions is computed.

At block 2123, analysis regions each with a phase difference conformingto a preset condition are merged into the focusing area.

Referring to FIG. 5, based on the embodiments described with referenceto FIG. 1 and FIG. 4, in an embodiment, when there is no target objectin the focusing area, the image processing method further includes thefollowing.

At block 217, it is determined whether a brightness of the focusing areais less than or equal to a first brightness threshold, whether a greenproportion is less than or equal to a proportion threshold and whether aspace frequency is less than or equal to a frequency threshold.

When at least one of the following satisfies: the brightness of thefocusing area being less than or equal to the first brightnessthreshold, the green proportion being less than or equal to theproportion threshold and the space frequency being less than or equal tothe frequency threshold, the act at block 216 is executed.

In some implementations, features of a landscape image include abrightness, a green proportion and a space frequency. Since a landscapeimage is captured in outdoor environment and there are generally trees,flowers or the like and enough light in the outdoor environment, animage can be determined as a landscape image when the image has a highbrightness, a high green proportion and a high space frequency.Therefore, by comparing each feature with a threshold thereof, it may bedetermined whether an image is a landscape image.

When it is determined that an image is not a landscape image, itindicates that there is not much requirement on definition of the image,such that it is only needed to convert the merged image into a mergedtrue-color image without improving the definition of the image.

The space frequency is an important factor for determine how to processthe image. The higher the space frequency is, the higher the requirementcan be set on the definition of the image. The lower the space frequencyis, the more the possibility that the merged image converted into amerged true-color image without improving the definition of the image.

In some implementations, a user may set the brightness threshold, theproportion threshold and the frequency threshold according to differentenvironments. Accordingly, the user may set one or more thresholdsaccording to the different environments to achieve an ideal shootingeffect.

In some implementations, the brightness threshold, the proportionthreshold and the frequency threshold can be stored in a memory of theelectronic device for user's selection. However, the present disclosureis not limited thereto.

Referring to FIG. 6, based on the embodiment described with reference toFIG. 5, in an embodiment, when the brightness of the focusing area isgreater than the first brightness threshold, the green proportion isgreater than the proportion threshold and the space frequency is greaterthan the frequency threshold, the image processing method furtherincludes the following.

At block 219, the image sensor outputs a color-block image.

The color-block image includes image pixel units arranged in a presetarray, each image pixel unit includes a plurality of original pixels,each photosensitive pixel unit corresponds to one image pixel unit. Inat least one embodiment, there is a one-to-one correspondence betweenthe photosensitive pixel units and the image pixel units. Eachphotosensitive pixel corresponds to one original pixel. In at least oneembodiment, there is a one-to-one correspondence between thephotosensitive pixels and the original pixels.

At block 220, the color-block image is converted into a simulation imageusing a first interpolation algorithm.

The simulation image includes simulation pixels arranged in an array,and each photosensitive pixel corresponds to one simulation pixel. In atleast one embodiment, there is a one-to-one correspondence between thephotosensitive pixels and the simulation pixels.

At block 230, the simulation image is converted into the simulationtrue-color image.

Since the landscape image requires a high image quality and an ambientbrightness of the landscape image is high, when a landscape image isdetected, the image sensor 10 is controlled to output a color-blockimage and the color-block image is converted into a simulation image andthen the simulation image is converted into a simulation true-colorimage.

Referring to FIG. 7, in some implementations, the act at block 220includes the following.

At block 221, it is determined whether a color of a simulation pixel isidentical to that of an original pixel at a same position as thesimulation pixel, if yes, an act at block 222 is executed, otherwise, anact at block 223 is executed.

At block 222, a pixel value of the original pixel is determined as apixel value of the simulation pixel.

At block 223, the pixel value of the simulation pixel is determinedaccording to a pixel value of an association pixel.

The association pixel is selected from an image pixel unit with a samecolor as the simulation pixel and adjacent to an image pixel unitincluding the original pixel.

FIG. 8 is a schematic diagram illustrating a circuit of an image sensoraccording to an embodiment of the present disclosure. FIG. 9 is aschematic diagram of an array of filter units according to an embodimentof the present disclosure. FIGS. 2-3 and 8-9 are best viewed together.

Referring to FIGS. 2-3 and 8-9, the image sensor 10 according to anembodiment of the present disclosure includes an array 12 ofphotosensitive pixel units and an array 14 of filter units arranged onthe array 12 of photosensitive pixel units.

Further, the array 12 of photosensitive pixel units includes a pluralityof photosensitive pixel units 12 a. Each photosensitive pixel unit 12 aincludes a plurality of adjacent photosensitive pixels 122. Eachphotosensitive pixel 122 includes a photosensitive element 1222 and atransmission tube 1224. The photosensitive element 1222 may be aphotodiode, and the transmission tube 1224 may be a MOS transistor.

The array 14 of filter units includes a plurality of filter units 14 a.Each filter unit 14 a corresponds to one photosensitive pixel unit 12 a.

In detail, in some examples, the filter units are arranged in a Bayerarray. In other words, four adjacent filter units 14 a include one redfilter unit, one blue filter unit and two green filter units.

Each photosensitive pixel unit 12 a corresponds to a filter unit 14 awith a same color. If a photosensitive pixel unit 12 a includes nadjacent photosensitive elements 1222, one filter unit 14 a covers nphotosensitive elements 1222 in one photosensitive pixel unit 12 a. Thefilter unit 14 a may be formed integrally, or may be formed byassembling n separate sub filters.

In some implementations, each photosensitive pixel unit 12 a includesfour adjacent photosensitive pixels 122. Two adjacent photosensitivepixels 122 collectively form one photosensitive pixel subunit 120. Thephotosensitive pixel subunit 120 further includes a source follower 124and an analog-to-digital converter 126. The photosensitive pixel unit 12a further includes an adder 128. A first electrode of each transmissiontube 1224 in the photosensitive pixel subunit 120 is coupled to acathode electrode of a corresponding photosensitive element 1222. Secondelectrodes of all the transmission tubes 1224 are collectively coupledto a gate electrode of the source follower 124 and coupled to ananalog-to-digital converter 126 via the source electrode of the sourcefollower 124. The source follower 124 may be a MOS transistor. Twophotosensitive pixel subunits 120 are coupled to the adder 128 viarespective source followers 124 and respective analog-to-digitalconverters 126.

In other words, four adjacent photosensitive elements 1222 in onephotosensitive pixel unit 12 a of the image sensor 10 according to anembodiment of the present disclosure collectively use one filter unit 14a with a same color as the photosensitive pixel unit. Eachphotosensitive element 1222 is coupled to a transmission tube 1224correspondingly. Two adjacent photosensitive elements 1222 collectivelyuse one source follower 124 and one analog-digital converter 126. Fouradjacent photosensitive elements 1222 collectively use one adder 128.

Further, four adjacent photosensitive elements 1222 are arranged in a2-by-2 array. Two photosensitive elements 1222 in one photosensitivepixel subunit 120 can be in a same row.

During an imaging process, when two photosensitive pixel subunits 120 orfour photosensitive elements 1222 covered by a same filter unit 14 a areexposed simultaneously, pixels can be merged, and the merged image canbe outputted.

In detail, the photosensitive element 1222 is configured to convertlight into charge, and the charge is proportional to an illuminationintensity. The transmission tube 1224 is configured to control a circuitto turn on or off according to a control signal. When the circuit isturned on, the source follower 124 is configured to convert the chargegenerated through light illumination into a voltage signal. Theanalog-to-digital converter 126 is configured to convert the voltagesignal into a digital signal. The adder 128 is configured to add twodigital signals for outputting.

Referring to FIG. 10, take an image sensor 10 of 16M as an example. Theimage sensor 10 according to an embodiment of the present disclosure canmerge photosensitive pixels 122 of 16M into photosensitive pixels of 4M,i.e., the image sensor 10 outputs the merged image. After the merging,the photosensitive pixel 122 quadruples in size, such that thephotosensibility of the photosensitive pixel 122 is increased. Inaddition, since most part of noise in the image sensor 10 is random,there may be noise points at one or two pixels. After fourphotosensitive pixels 122 are merged into a big photosensitive pixel122, an effect of noise points on the big photosensitive pixel isreduced, i.e., the noise is weakened and SNR (signal to noise ratio) isimproved.

However, when the size of the photosensitive pixel 122 is increased, thepixel value is decreased, and thus the resolution of the merged image isdecreased.

During an imaging process, when four photosensitive elements 1222covered by a same filter unit 14 a are exposed in sequence, acolor-block image is output.

In detail, the photosensitive element 1222 is configured to convertlight into charge, and the charge is proportional to an illuminationintensity. The transmission tube 1224 is configured to control a circuitto turn on or off according to a control signal. When the circuit isturned on, the source follower 124 is configured to convert the chargegenerated through light illumination into a voltage signal. Theanalog-to-digital converter 126 is configured to convert the voltagesignal into a digital signal.

Referring to FIG. 11, take an image sensor 10 of 16M as an example. Theimage sensor according to an embodiment of the present disclosure canoutput photosensitive pixels 122 of 16M, i.e., the image sensor 200outputs the color-block image. The color-block image includes imagepixel units. The image pixel unit includes original pixels arranged in a2-by-2 array. The size of the original pixel is the same as that of thephotosensitive pixel 122. However, since a filter unit 14 a coveringfour adjacent photosensitive elements 1222 has a same color (i.e.,although four photosensitive elements 1222 are exposed respectively, thefilter unit 14 a covering the four photosensitive elements has a samecolor), four adjacent original pixels in each image pixel unit of theoutput image have a same color, and thus the resolution of the imagecannot be increased.

The image processing method according to an embodiment of the presentdisclosure is able to process the output color-block image to obtain asimulation image.

In some embodiments, when a merged image is output, four adjacentphotosensitive pixels 122 with the same color can be output as onemerged pixel. Accordingly, four adjacent merged pixels in the mergedimage can be considered as being arranged in a typical Bayer array, andcan be processed directly to output a merged true-color image. When acolor-block image is output, each photosensitive pixel 122 is outputseparately. Since four adjacent photosensitive pixels 122 have a samecolor, four adjacent original pixels in an image pixel unit have a samecolor, which form an untypical Bayer array. However, the untypical Bayerarray cannot be directly processed. In other words, when the imagesensor 10 adopts a same apparatus for processing the image, in order torealize a compatibility of the true-color image outputs under two modes(i.e., the merged true-color image under a merged mode and thesimulation true-color image under a color-block mode), it is required toconvert the color-block image into the simulation image, or to convertthe image pixel unit in an untypical Bayer array into pixels arranged inthe typical Bayer array.

The simulation image includes simulation pixels arranged in the Bayerarray. Each photosensitive pixel corresponds to one simulation pixel.One simulation pixel in the simulation image corresponds to an originalpixel located at the same position as the simulation pixel and in thecolor-block image.

Referring to FIG. 12, for the simulation pixels R3′3′ and R5′5′, thecorresponding original pixels are R33 and B55.

When the simulation pixel R3′3′ is obtained, since the simulation pixelR3′3′ has the same color as the corresponding original pixel R33, thepixel value of the original pixel R33 is directly determined as thepixel value of the simulation pixel R3′3′ during conversion.

When the simulation pixel R5′5′ is obtained, since the simulation pixelR5′5′ has a color different from that of the corresponding originalpixel B55, the pixel value of the original pixel B55 cannot be directlydetermined as the pixel value of the simulation pixel R5′5′, and it isrequired to calculate the pixel value of the simulation pixel R5′5′according to an association pixel of the simulation pixel R5′5′ by aninterpolation algorithm.

It should be noted that, a pixel value of a pixel mentioned in thecontext should be understood in a broad sense as a color attribute valueof the pixel, such as a color value.

There may be more than one association pixel unit for each simulationpixel, for example, there may be four association pixel units, in whichthe association pixel units have the same color as the simulation pixeland are adjacent to the image pixel unit including the original pixel atthe same position as the simulation pixel.

It should be noted that, “adjacent” here should be understood in a broadsense. Taking FIG. 12 as an example, the simulation pixel R5′5′corresponds to the original pixel B55. The image pixel units 400, 500,600 and 700 are selected as the association pixel units, but other redimage pixel units far away from the image pixel unit where the originalpixel B55 is located are not selected as the association pixel units. Ineach association pixel unit, the red original pixel closest to theoriginal pixel B55 is selected as the association pixel, which meansthat the association pixels of the simulation pixel R5′5′ include theoriginal pixels R44, R74, R47 and R77. The simulation pixel R5′5′ isadjacent to and has the same color as the original pixels R44, R74, R47and R77.

In different cases, the original pixels can be converted into thesimulation pixels in different ways, thus converting the color-blockimage into the simulation image. Since the filters in the Bayer arrayare adopted when shooting the image, the SNR of the image is improved.During the image processing procedure, the interpolation processing isperformed on the color-block image by the interpolation algorithm, suchthat the distinguishability and resolution of the image can be improved.

Referring to FIG. 13, in some implementations, the act at block 223(i.e., determining the pixel value of the simulation pixel according tothe pixel value of the association pixel) includes the following.

At block 2232, a change of the color of the simulation pixel in eachdirection of at least two directions is calculated according to thepixel value of the association pixel.

At block 2234, a weight in each direction of the at least two directionsis calculated according to the change.

At block 2236, the pixel value of the simulation pixel is calculatedaccording to the weight and the pixel value of the association pixel.

In detail, the interpolation processing is realized as follows: withreference to energy changes of the image in different directions andaccording to weights of the association pixels in different directions,the pixel value of the simulation pixel is calculated by a linearinterpolation. From the direction having a smaller energy change, it canget a higher reference value, i.e., the weight for this direction in theinterpolation is high.

In some examples, for sake of convenience, only the horizontal directionand the vertical direction are considered.

The pixel value of the simulation pixel R5′5′ is obtained by aninterpolation based on the original pixels R44, R74, R47 and R77. Sincethere is no original pixel with a same color as the simulation pixel(i.e., R) in the horizontal direction and the vertical direction of theoriginal pixel B55 corresponding the simulation pixel R5′5′, a componentof this color (i.e., R) in each of the horizontal direction and thevertical direction is calculated according to the association pixels.The components in the horizontal direction are R45 and R75, thecomponents in the vertical direction are R54 and R57. All the componentscan be calculated according to the original pixels R44, R74, R47 andR77.

In detail, R45=R44*⅔+R47*⅓, R75=⅔*R74+⅓*R77, R54=⅔*R44+⅓*R74,R57=⅔*R47+⅓*R77.

The change of color and the weight in each of the horizontal directionand the vertical direction are calculated respectively. In other words,according to the change of color in each direction, the reference weightin each direction used in the interpolation is determined. The weight inthe direction with a small change is high, while the weight in thedirection with a big change is low. The change in the horizontaldirection is X1=|R45−R75|. The change in the vertical direction isX2=|R54−R57|, W1=X1/(X1+X2), W2=X2/(X1+X2).

After the above calculation, the pixel value of the simulation pixelR5′5′ can be calculated as R5′5′=(⅔*R45+⅓*R75)*W2+(⅔*R54+⅓*R57)*W1. Itcan be understood that, if X1>X2, then W1>W2. The weight in thehorizontal direction is W2, and the weight in the vertical direction isW1, vice versa.

Accordingly, the pixel value of the simulation pixel can be calculatedby the interpolation algorithm. After the calculations on theassociation pixels, the original pixels can be converted into thesimulation pixels arranged in the typical Bayer array. In other words,four adjacent simulation pixels arranged in the 2-by-2 array include onered simulation pixel, two green simulation pixels and one bluesimulation pixel.

It should be noted that, the interpolation processing is not limited tothe above-mentioned method, in which only the pixel values of pixelswith a same color as the simulation pixel in the vertical direction andthe horizontal direction are considered during calculating the pixelvalue of the simulation pixel. In other embodiments, pixel values ofpixels with other colors can also be considered.

Referring to FIG. 14, in some embodiments, before the act at block 223,the method further includes performing a white-balance compensation onthe color-block image, as illustrated at block 224.

Accordingly, after the act at 223, the method further includesperforming a reverse white-balance compensation on the simulation image,as illustrated at block 225.

In detail, in some examples, when converting the color-block image intothe simulation image, during the interpolation, the red and bluesimulation pixels not only refer to the color weights of original pixelshaving the same color as the simulation pixels, but also refer to thecolor weights of original pixels with the green color. Thus, it isrequired to perform the white-balance compensation before theinterpolation to exclude an effect of the white-balance in theinterpolation calculation. In order to avoid the white-balance of thecolor-block image, it is required to perform the reverse white-balancecompensation after the interpolation according to gain values of thered, green and blue colors in the compensation.

In this way, the effect of the white-balance in the interpolationcalculation can be excluded, and the simulation image obtained after theinterpolation can keep the white-balance of the color-block image.

Referring to FIG. 14 again, in some implementations, before the act atblock 223, the method further includes performing a bad-pointcompensation on the color-block image, as illustrated at block 226.

It can be understood that, limited by the manufacturing process, theremay be bad points in the image sensor 10. The bad point presents a samecolor all the time without varying with the photosensibility, whichaffects quality of the image. In order to ensure an accuracy of theinterpolation and prevent from the effect of the bad points, it isrequired to perform the bad-point compensation before the interpolation.

In detail, during the bad-point compensation, the original pixels aredetected. When an original pixel is detected as the bad point, thebad-point compensation is performed according to pixel values of otheroriginal pixels in the image pixel unit where the original pixel islocated.

In this way, the effect of the bad point on the interpolation can beavoided, thus improving the quality of the image.

Referring to FIG. 14 again, in some implementations, before the act atblock 223, the method includes performing a crosstalk compensation onthe color-block image, as illustrated at block 227.

In detail, four photosensitive pixels 122 in one photosensitive pixelunit 12 a cover the filters with the same color, and the photosensitivepixels 122 have differences in photosensibility, such that fixedspectrum noise may occur in pure-color areas in the simulationtrue-color image outputted after converting the simulation image and thequality of the image may be affected. Therefore, it is required toperform the crosstalk compensation.

In some implementations, compensation parameters can be set by:providing a preset luminous environment, configuring imaging parametersof the imaging apparatus, capturing multi-frame images, processing themulti-frame images to obtain crosstalk compensation parameters, andstoring the crosstalk compensation parameters.

As explained above, in order to perform the crosstalk compensation, itis required to obtain the compensation parameters during themanufacturing process of the image sensor of the imaging apparatus, andto store the parameters related to the crosstalk compensation into thestorage of the imaging apparatus or the electronic device provided withthe imaging apparatus, such as the mobile phone or tablet computer.

The preset luminous environment, for example, may include an LED uniformplate having a color temperature of about 5000 K and a brightness ofabout 1000 lux. The imaging parameters may include a gain value, ashutter value and a location of a lens. After setting the relatedparameters, the crosstalk compensation parameters can be obtained.

During the process, multiple color-block images are obtained using thepreset imaging parameters in the preset luminous environment, andcombined into one combination color-block image, such that the effect ofnoise caused by using a single color-block image as reference can bereduced.

Referring to FIG. 15, take the image pixel unit Gr as an example. Theimage pixel unit Gr includes original pixels Gr1, Gr2, Gr3 and Gr4. Thepurpose of the crosstalk compensation is to adjust the photosensitivepixels which may have different photosensibilities to have the samephotosensibility. An average pixel value of the image pixel unit isGr_avg=(Gr1+Gr2+Gr3+Gr4)/4, which represents an average level ofphotosensibilities of the four photosensitive pixels. By configuring theaverage value as a reference value, ratios of Gr1/Gr_avg, Gr2/Gr_avg,Gr3/Gr_avg and Gr4/Gr_avg are calculated. It can be understood that, bycalculating a ratio of the pixel value of each original pixel to theaverage pixel value of the image pixel unit, a deviation between eachoriginal pixel and the reference value can be reflected. Four ratios canbe recorded in a storage of a related device as the compensationparameters, and can be retrieved during the imaging process tocompensate for each original pixel, thus reducing the crosstalk andimproving the quality of the image.

Generally, after setting the crosstalk compensation parameters,verification is performed on the parameters to determine the accuracy ofthe parameters.

During the verification, a color-block image is obtained with the sameluminous environment and same imaging parameters as the preset luminousenvironment and the preset imaging parameters, and the crosstalkcompensation is performed on the color-block image according to thecalculated compensation parameters to calculate compensated Gr′_avg,Gr′1/Gr′_avg, Gr′2/Gr′_avg, Gr′3/Gr′_avg and Gr′4/Gr′_avg. The accuracyof parameters can be determined according to the calculation resultsfrom a macro perspective and a micro perspective. From the microperspective, when a certain original pixel after the compensation stillhas a big deviation which is easy to be sensed by the user after theimaging process, it means that the parameters are not accurate. From themacro perspective, when there are too many original pixels withdeviations after the compensation, the deviations as a whole can besensed by the user even if a single original pixel has a smalldeviation, and in this case, the parameters are also not accurate. Thus,a ratio threshold can be set for the micro perspective, and anotherratio threshold and a number threshold can be set for the macroperspective. In this way, the verification can be performed on thecrosstalk compensation parameters to ensure the accuracy of thecompensation parameters and to reduce the effect of the crosstalk on thequality of the image.

Referring to FIG. 16, in some implementations, after the act at block223, the method further includes performing at least one of a mirrorshape correction, a demosaicking processing, a denoising processing andan edge sharpening processing on the simulation image, as illustrated atblock 228.

It can be understood that, after the color-block image is converted intothe simulation image, the simulation pixels are arranged in the typicalBayer array. The simulation image can be processed, during which, themirror shape correction, the demosaicking processing, the denoisingprocessing and the edge sharpening processing are included, such thatthe simulation true-color image can be obtained and output to the user.

Referring to FIG. 17, in some implementations, the act at block 216includes the following.

At block 2162, the merged image is converted into a restoration imageusing a second interpolation algorithm.

The restoration image includes restoration pixels arranged in an array,and each photosensitive pixel corresponds to one restoration pixel. Acomplexity of the second interpolation algorithm is less than that ofthe first interpolation algorithm.

At block 2164, the restoration image is converted into the mergedtrue-color image.

In some implementations, the algorithm complexity includes the timecomplexity and the space complexity, and both the time complexity andthe space complexity of the second interpolation algorithm are less thanthose of the first interpolation algorithm. The time complexity isconfigured to measure a time consumed by the algorithm, and the spacecomplexity is configured to measure a storage space consumed by thealgorithm. If the time complexity is small, it indicates that thealgorithm consumes little time. If the space complexity is small, itindicates that the algorithm consumes little storage space. Thus, it isadvantageous to improve calculation speed by using the secondinterpolation algorithm, such that the shooting process is smooth, thusimproving the user experience.

In some implementations, the second interpolation algorithm is used toquadruple the merged image without other complicated calculations, suchthat the restoration image corresponding to the simulation image can beobtained.

It can be understood that, after obtaining the simulation true-colorimage, the denoising processing and the edge sharpening processing areperformed on the simulation true-color image. Thus, the simulationtrue-color image with high quality can be obtained after the processingand output to the user.

In another aspect, the present disclosure also provides an imageprocessing apparatus.

FIG. 18 is a block diagram of an image processing apparatus according toan embodiment of the present disclosure. Referring to FIG. 18 and FIGS.2-3 and 8-9, an image processing apparatus 4000 is illustrated. Theimage processing apparatus 4000 is applied in an electronic device. Theelectronic device includes an imaging apparatus including an imagesensor 10. As illustrated above, the image sensor 10 includes an array12 of photosensitive pixel units and an array 14 of filter unitsarranged on the array 12 of photosensitive pixel units. Each filter unit14 a corresponds to one photosensitive pixel unit 12 a, and eachphotosensitive pixel unit 12 a includes a plurality of photosensitivepixels 122. The image processing apparatus 4000 includes anon-transitory computer-readable medium 4600 and an instructionexecution system 4800. The non-transitory computer-readable medium 4600includes computer-executable instructions stored thereon. As illustratedin FIG. 18, the non-transitory computer-readable medium 4600 includes aplurality of program modules, including a first control module 411, afirst determining module 412, a second determining module 414, a firstconverting module 416. The instruction execution system 4800 isconfigured by the instructions stored in the medium 4600 to implementthe program modules.

The first control module 411 is configured to output a merged image bythe image sensor 10. The merged image includes an array of mergedpixels, and a plurality of photosensitive pixels 122 in a samephotosensitive pixel unit 12 a are collectively outputted as one mergedpixel. The first determining module 412 is configured to determine afocusing area of the merged image. The second determining module 414 isconfigured to determine whether there is a target object in the focusingarea. The first converting module 416 is configured to convert themerged image into a merged true-color image when there is the targetobject in the focusing area.

In other words, the act at block 211 can be implemented by the firstcontrol module 411. The act at block 212 can be implemented by the firstdetermining module 412. The act at block 214 can be implemented by thesecond determining module 414. The act at block 216 can be implementedby the first converting module 416.

With the image processing apparatus according to embodiments of thepresent disclosure, the image sensor can be controlled to output asuitable image by identifying and determining the image in depth offocus, a situation that it takes too much work to output a high qualityimage by the image sensor can be avoided, thus reducing work time of theelectronic device, improving work efficiency and improving the userexperience.

Referring to FIG. 19, the first determining module 412 includes adividing unit 4121, a computing unit 4122, and a merging unit 4123. Thedividing unit 4121 is configured to divide the color-block image into aplurality of analysis regions arranged in an array. The computing unit4122 is configured to compute a phase difference of each of theplurality of analysis regions. The merging unit 4123 is configured tomerge analysis regions each with a phase difference conforming to apreset condition into the focusing area.

In other words, the act at block 2121 can be implemented by the dividingunit 4121. The act at block 2122 can be implemented by the computingunit 4122. The act at block 2123 can be implemented by the merging unit4123.

FIG. 20 is a block diagram of an image processing apparatus according toanother embodiment of the present disclosure. Referring to FIG. 20,based on the embodiment described with reference to FIG. 18, in anembodiment, the image processing apparatus 400 further includes a thirddetermining module 417.

a second judging module 418, a second acquiring module 419, a secondconverting module 420, and a third converting module 430.

The third determining module 417 is configured to determine whether abrightness of the focusing area is less than or equal to a firstbrightness threshold, whether a green proportion is less than or equalto a proportion threshold and whether a space frequency is less than orequal to a frequency threshold when there is no target object in thefocusing area. The first converting module 416 is further configured toconvert the merged image into the merged true-color image when at leastone of the following satisfies: the brightness of the focusing areabeing less than or equal to the first brightness threshold, the greenproportion being less than or equal to the proportion threshold and thespace frequency being less than or equal to the frequency threshold.

Referring FIG. 20 again, in an embodiment, the image processingapparatus 400 further includes a second control module 419, a secondconverting module 420 and a third converting module 430.

The second control module 419 is configured to output a color-blockimage by the image sensor when the brightness of the focusing area isgreater than the first brightness threshold, the green proportion isgreater than the proportion threshold and the space frequency is greaterthan the frequency threshold. The color-block image includes image pixelunits arranged in a preset array, and each image pixel unit includes aplurality of original pixels. Each photosensitive pixel unit correspondsto one image pixel unit, and each photosensitive pixel corresponds toone original pixel.

The second converting module 420 is configured to convert thecolor-block image into a simulation image using a first interpolationalgorithm. The simulation image includes simulation pixels arranged inan array and each photosensitive pixel corresponds to one simulationpixel.

The third converting module 430 is configured to convert the simulationimage into a simulation true-color image.

In other words, the act at block 217 can be implemented by the thirddetermining module 417. The act at block 219 can be implemented by thesecond control module 419. The act at block 220 can be implemented bythe second converting module 420. The act at block 230 can beimplemented by the third converting module 430.

Referring to FIG. 21, the second converting module 420 includes a firstdetermining unit 421, a second determining unit 422, and a thirddetermining unit 423. The first determining unit 421 is configured todetermine whether a color of a simulation pixel is identical to that ofan original pixel at a same position as the simulation pixel. The seconddetermining unit 422 is configured to determine a pixel value of theoriginal pixel as a pixel value of the simulation pixel when the colorof the simulation pixel is identical to that of the original pixel atthe same position as the simulation pixel. The third determining unit423 is configured to determine the pixel value of the simulation pixelaccording to pixel values of association pixels when the color of thesimulation pixel is different from that of the original pixel at thesame position as the simulation pixel. The association pixels areselected from an image pixel unit with a same color as the simulationpixel and adjacent to the image pixel unit including the original pixel.

In other words, the act at block 221 can be implemented by the firstdetermining unit 421. The act at block 222 can be implemented by thesecond determining unit 422. The act at block 223 can be implemented bythe third determining unit 423.

Referring to FIG. 22, in some implementations, the third determiningunit 423 includes a first calculating subunit 4232, a second calculatingsubunit 4234 and a third calculating subunit 4236. The act at block 2232can be implemented by the first calculating subunit 4232. The act atblock 2234 can be implemented by the second calculating subunit 4234.The act at block 2236 can be implemented by the third calculatingsubunit 4236. In other words, the first calculating subunit 4232 isconfigured to calculate a change of the color of the simulation pixel ineach direction of at least two directions according to the pixel valueof the association pixel. The second calculating subunit 4234 isconfigured to calculate a weight in each direction of the at least twodirections according to the change. The third calculating subunit 4236is configured to calculate the pixel value of the simulation pixelaccording to the weight and the pixel value of the association pixel.

FIG. 23 is a block diagram of a second converting module according toanother embodiment of the present disclosure. Referring to FIG. 23, insome implementations, the second converting module 420 further includesa first compensating unit 424 and a restoring unit 425. The act at block224 can be implemented by the first compensating unit 424. The act atblock 225 can be implemented by the restoring unit 425. In other words,the first compensating unit 424 is configured to perform a white-balancecompensation on the color-block image. The restoring unit 425 isconfigured to perform a reverse white-balance compensation on thesimulation image.

In some implementations, the second converting module 420 furtherincludes a second compensating unit 426. The act at block 226 can beimplemented by the second compensating unit 426. In other words, thesecond compensating unit 426 is configured to perform a bad-pointcompensation on the color-block image.

In some implementations, the second converting module 420 furtherincludes a third compensating unit 427. The act at block 227 can beimplemented by the third compensating unit 427. In other words, thethird compensating unit 427 is configured to perform a crosstalkcompensation on the color-block image.

FIG. 24 is a block diagram of a second converting module according toanother embodiment of the present disclosure. Referring to FIG. 24, insome implementations, the second converting module 420 includes aprocessing unit 428. The act at block 228 can be implemented by theprocessing unit 428. In other words, the processing unit 428 isconfigured to perform at least one of a mirror shape correction, ademosaicking processing, a denoising processing and an edge sharpeningprocessing on the simulation image.

Referring to FIG. 25, in some implementations, the first convertingmodule 416 includes a first converting unit 4162 and a second convertingunit 4164. The first converting unit 4162 is configured to convert themerged image into a restoration image using a second interpolationalgorithm. The restoration image includes restoration pixels arranged inan array, and each photosensitive pixel corresponds to one restorationpixel. A complexity of the second interpolation algorithm is less thanthat of the first interpolation algorithm. The second converting unit4164 is configured to convert the restoration image into the mergedtrue-color image. In other words, the act at block 2162 is implementedby the first converting unit 4162. The act at block 2164 is implementedby the second converting unit 4164.

The present disclosure also provides an electronic device.

FIG. 26 is a block diagram of an electronic device 1000 according to anembodiment of the present disclosure. Referring to FIG. 26, theelectronic device 1000 of the present disclosure includes a housing1001, a processor 1002, a memory 1003, a circuit board 1006, a powersupply circuit 1007 and an imaging apparatus 100, The circuit board 1006is enclosed by the housing 1001. The processor 1002 and the memory 1003are positioned on the circuit board 1006. The power supply circuit 1007is configured to provide power for respective circuits or components ofthe electronic device 1000. The memory 1003 is configured to storeexecutable program codes. The imaging apparatus 100 includes an imagesensor 10. As illustrated above, the image sensor 10 includes an array12 of photosensitive pixel units and an array 14 of filter unitsarranged on the array 12 of photosensitive pixel units. Each filter unit14 a corresponds to one photosensitive pixel unit 12 a, and eachphotosensitive pixel unit 12 a includes a plurality of photosensitivepixels 122.

The processor 1002 is configured to run a program corresponding to theexecutable program codes by reading the executable program codes storedin the memory 1003, to perform following operations: outputting a mergedimage by the image sensor, in which, the merged image includes an arrayof merged pixels, and a plurality of photosensitive pixels in a samephotosensitive pixel unit are collectively output as one merged pixel;determining a focusing area of the merged image; determining whetherthere is a target object in the focusing area; and when there is thetarget object in the focusing area, converting the merged image into amerged true-color image.

In some implementations, the imaging apparatus includes a front cameraor a real camera (not illustrated in FIG. 26).

In some implementations, the processor 1002 is configured to run aprogram corresponding to the executable program codes by reading theexecutable program codes stored in the memory 1003, to determine afocusing area of the merged image by acts of: dividing the merged imageinto a plurality of analysis regions arranged in an array; computing aphase difference of each of the plurality of analysis regions; andmerging analysis regions each with a phase difference conforming to apreset condition into the focusing area.

In some implementations, the processor 1002 is configured to run aprogram corresponding to the executable program codes by reading theexecutable program codes stored in the memory 1003, to perform followingoperation: when there is no target object in the focusing area,determining whether a brightness of the focusing area is less than orequal to a first brightness threshold, whether a green proportion of thefocusing area is less than or equal to a proportion threshold andwhether a space frequency of the focusing area is less than or equal toa frequency threshold; and when at least one of the following satisfies:the brightness of the focusing area being less than or equal to thefirst brightness threshold, the green proportion being less than orequal to the proportion threshold and the space frequency being lessthan or equal to the frequency threshold, converting the merged imageinto the merged true-color image.

In some implementations, the processor 1002 is configured to run aprogram corresponding to the executable program codes by reading theexecutable program codes stored in the memory 1003, to perform followingoperation: when the brightness of the focusing area is greater than thefirst brightness threshold, the green proportion is greater than theproportion threshold and the space frequency is greater than thefrequency threshold, outputting a color-block image by the image sensor,in which, the color-block image includes image pixel units arranged in apreset array, each image pixel unit includes a plurality of originalpixels, each photosensitive pixel unit corresponds to one image pixelunit, and each photosensitive pixel corresponds to one original pixel;converting the color-block image into a simulation image using a firstinterpolation algorithm, in which, the simulation image includessimulation pixels arranged in an array, and each photosensitive pixelcorresponds to one simulation pixel; and converting the simulation imageinto the simulation true-color image.

In some implementations, the processor 1002 is configured to run aprogram corresponding to the executable program codes by reading theexecutable program codes stored in the memory 1003, to convert thecolor-block image into the simulation image using the firstinterpolation algorithm by acts of: determining whether a color of asimulation pixel is identical to that of an original pixel at a sameposition as the simulation pixel; when the color of the simulation pixelis identical to that of the original pixel at the same position as thesimulation pixel, determining a pixel value of the original pixel as apixel value of the simulation pixel; and when the color of thesimulation pixel is different from that of the original pixel at thesame position as the simulation pixel, determining the pixel value ofthe simulation pixel according to a pixel value of an association pixel,in which the association pixel is selected from an image pixel unit witha same color as the simulation pixel and adjacent to an image pixel unitincluding the original pixel.

In some implementations, the processor 1002 is configured to run aprogram corresponding to the executable program codes by reading theexecutable program codes stored in the memory 1003, to determine thepixel value of the simulation pixel according to a pixel value of anassociation pixel by acts of: calculating a change of the color of thesimulation pixel in each direction of at least two directions accordingto the pixel value of the association pixel; calculating a weight ineach direction of the at least two directions according to the change;and calculating the pixel value of the simulation pixel according to theweight and the pixel value of the association pixel.

In some implementations, the processor 1002 is configured to run aprogram corresponding to the executable program codes by reading theexecutable program codes stored in the memory 1003, to perform followingoperations: performing a white-balance compensation on the color-blockimage; and performing a reverse white-balance compensation on thesimulation image.

In some implementations, the processor 1002 is configured to run aprogram corresponding to the executable program codes by reading theexecutable program codes stored in the memory 1003, to perform followingoperation: performing at least one of a bad-point compensation and acrosstalk compensation on the color-block image.

In some implementations, the processor 1002 is configured to run aprogram corresponding to the executable program codes by reading theexecutable program codes stored in the memory 1003, to perform followingoperations: performing at least one of a mirror shape correction, ademosaicking processing, a denoising processing and an edge sharpeningprocessing on the simulation image.

In some implementations, the processor 1002 is configured to run aprogram corresponding to the executable program codes by reading theexecutable program codes stored in the memory 1003, to convert themerged image into a merged true-color image by acts of: converting themerged image into a restoration image using a second interpolationalgorithm, in which the restoration image includes restoration pixelsarranged in an array, each photosensitive pixel corresponds to onerestoration pixel, and a complexity of the second interpolationalgorithm is less than that of the first interpolation algorithm; andconverting the restoration image into the merged true-color image.

In some implementations, the electronic device may be an electronicequipment provided with an imaging apparatus, such as a mobile phone ora tablet computer, which is not limited herein.

The electronic device 1000 may further include an inputting component(not illustrated in FIG. 24). It should be understood that, theinputting component may further include one or more of the following: aninputting interface, a physical button of the electronic device 1000, amicrophone, etc.

It should be understood that, the electronic device 1000 may furtherinclude one or more of the following components (not illustrated in FIG.24): an audio component, an input/output (I/O) interface, a sensorcomponent and a communication component. The audio component isconfigured to output and/or input audio signals, for example, the audiocomponent includes a microphone. The I/O interface is configured toprovide an interface between the processor 1002 and peripheral interfacemodules. The sensor component includes one or more sensors to providestatus assessments of various aspects of the electronic device 1000. Thecommunication component is configured to facilitate communication, wiredor wirelessly, between the electronic device 1000 and other devices.

It is to be understood that phraseology and terminology used herein withreference to device or element orientation (such as, terms like“center”, “longitudinal”, “lateral”, “length”, “width”, “height”, “up”,“down”, “front”, “rear”, “left”, “right”, “vertical”, “horizontal”,“top”, “bottom”, “inside”, “outside”, “clockwise”, “anticlockwise”,“axial”, “radial”, “circumferential”) are only used to simplifydescription of the present invention, and do not indicate or imply thatthe device or element referred to must have or operated in a particularorientation. They cannot be seen as limits to the present disclosure.

Moreover, terms of “first” and “second” are only used for descriptionand cannot be seen as indicating or implying relative importance orindicating or implying the number of the indicated technical features.Thus, the features defined with “first” and “second” may comprise orimply at least one of these features. In the description of the presentdisclosure, “a plurality of” means two or more than two, unlessspecified otherwise.

In the present disclosure, unless specified or limited otherwise, theterms “mounted,” “connected,” “coupled,” “fixed” and the like are usedbroadly, and may be, for example, fixed connections, detachableconnections, or integral connections; may also be mechanical orelectrical connections; may also be direct connections or indirectconnections via intervening structures; may also be inner communicationsof two elements or interactions of two elements, which can be understoodby those skilled in the art according to specific situations.

In the present disclosure, unless specified or limited otherwise, astructure in which a first feature is “on” a second feature may includean embodiment in which the first feature directly contacts the secondfeature, and may also include an embodiment in which the first featureindirectly contacts the second feature via an intermediate medium.Moreover, a structure in which a first feature is “on”, “over” or“above” a second feature may indicate that the first feature is rightabove the second feature or obliquely above the second feature, or justindicate that a horizontal level of the first feature is higher than thesecond feature. A structure in which a first feature is “below”, or“under” a second feature may indicate that the first feature is rightunder the second feature or obliquely under the second feature, or justindicate that a horizontal level of the first feature is lower than thesecond feature.

Various embodiments and examples are provided in the followingdescription to implement different structures of the present disclosure.In order to simplify the present disclosure, certain elements andsettings will be described. However, these elements and settings areonly examples and are not intended to limit the present disclosure. Inaddition, reference numerals may be repeated in different examples inthe disclosure. This repeating is for the purpose of simplification andclarity and does not refer to relations between different embodimentsand/or settings. Furthermore, examples of different processes andmaterials are provided in the present disclosure. However, it would beappreciated by those skilled in the art that other processes and/ormaterials may be also applied.

Reference throughout this specification to “an embodiment,” “someembodiments,” “an example,” “a specific example,” or “some examples,”means that a particular feature, structure, material, or characteristicdescribed in connection with the embodiment or example is included in atleast one embodiment or example of the present disclosure. In thisspecification, exemplary descriptions of aforesaid terms are notnecessarily referring to the same embodiment or example. Furthermore,the particular features, structures, materials, or characteristics maybe combined in any suitable manner in one or more embodiments orexamples. Moreover, those skilled in the art could combine differentembodiments or different characteristics in embodiments or examplesdescribed in the present disclosure.

Any process or method described in a flow chart or described herein inother ways may be understood to include one or more modules, segments orportions of codes of executable instructions for achieving specificlogical functions or steps in the process, and the scope of a preferredembodiment of the present disclosure includes other implementations,wherein the order of execution may differ from that which is depicted ordiscussed, including according to involved function, executingconcurrently or with partial concurrence or in the contrary order toperform the function, which should be understood by those skilled in theart.

The logic and/or step described in other manners herein or shown in theflow chart, for example, a particular sequence table of executableinstructions for realizing the logical function, may be specificallyachieved in any computer readable medium to be used by the instructionexecution system, device or equipment (such as the system based oncomputers, the system comprising processors or other systems capable ofacquiring the instruction from the instruction execution system, deviceand equipment and executing the instruction), or to be used incombination with the instruction execution system, device and equipment.As to the specification, “the computer readable medium” may be anydevice adaptive for including, storing, communicating, propagating ortransferring programs to be used by or in combination with theinstruction execution system, device or equipment. More specificexamples of the computer-readable medium comprise but are not limitedto: an electronic connection (an electronic device) with one or morewires, a portable computer enclosure (a magnetic device), a randomaccess memory (RAM), a read-only memory (ROM), an erasable programmableread-only memory (EPROM or a flash memory), an optical fiber device anda portable compact disk read-only memory (CDROM). In addition, thecomputer-readable medium may even be a paper or other appropriate mediumcapable of printing programs thereon, this is because, for example, thepaper or other appropriate medium may be optically scanned and thenedited, decrypted or processed with other appropriate methods whennecessary to obtain the programs in an electric manner, and then theprograms may be stored in the computer memories.

It should be understood that each part of the present disclosure may berealized by hardware, software, firmware or their combination. In theabove embodiments, a plurality of steps or methods may be realized bythe software or firmware stored in the memory and executed by theappropriate instruction execution system. For example, if it is realizedby the hardware, likewise in another embodiment, the steps or methodsmay be realized by one or a combination of the following techniquesknown in the art: a discrete logic circuit having a logic gate circuitfor realizing a logic function of a data signal, an application-specificintegrated circuit having an appropriate combination logic gate circuit,a programmable gate array (PGA), a field programmable gate array (FPGA),etc.

Those skilled in the art shall understand that all or parts of the stepsin the above exemplifying method for the present disclosure may beachieved by commanding the related hardware with programs, the programsmay be stored in a computer-readable storage medium, and the programscomprise one or a combination of the steps in the method embodiments ofthe present disclosure when running on a computer.

In addition, each function cell of the embodiments of the presentdisclosure may be integrated in a processing module, or these cells maybe separate physical existence, or two or more cells are integrated in aprocessing module. The integrated module may be realized in a form ofhardware or in a form of software function modules. When the integratedmodule is realized in a form of software function module and is sold orused as a standalone product, the integrated module may be stored in acomputer-readable storage medium.

The storage medium mentioned above may be read-only memories, magneticdisks, CD, etc.

Although embodiments of present disclosure have been shown and describedabove, it should be understood that above embodiments are justexplanatory, and cannot be construed to limit the present disclosure,for those skilled in the art, changes, alternatives, and modificationscan be made to the embodiments without departing from spirit, principlesand scope of the present disclosure.

What is claimed is:
 1. A control method, applied for controlling anelectronic device, wherein the electronic device comprises an imagingapparatus and a displayer, the imaging apparatus comprises an imagesensor, the image sensor comprises an array of photosensitive pixelunits and an array of filter units arranged on the array ofphotosensitive pixel units, each filter unit corresponds to onephotosensitive pixel unit, and each photosensitive pixel unit comprisesa plurality of photosensitive pixels, the control method comprises:controlling the image sensor to output a merged image, wherein, themerged image comprises an array of merged pixels, and a plurality ofphotosensitive pixels in each photosensitive pixel unit are collectivelyoutput as one merged pixel; dividing the merged image into a pluralityof analysis regions arranged in an array; computing a phase differenceof each of the plurality of analysis regions; merging analysis regionseach with a phase difference conforming to a preset condition into afocusing area; identifying whether there is a face in the focusing area;when there is the face in the focusing area, converting the merged imageinto a merged true-color image; when there is no face in the focusingarea, determining whether a brightness of the focusing area is less thanor equal to a brightness threshold, whether a green proportion of thefocusing area is less than or equal to a proportion threshold andwhether a space frequency of the focusing area is less than or equal toa frequency threshold; when the brightness of the focusing area is lessthan or equal to the brightness threshold, the green proportion is lessthan or equal to the proportion threshold and the space frequency isless than or equal to the frequency threshold, converting the mergedimage into the merged true-color image; when the brightness of thefocusing area is greater than the brightness threshold, the greenproportion is greater than the proportion threshold and the spacefrequency is greater than the frequency threshold, controlling the imagesensor to output a color-block image, wherein, the color-block imagecomprises image pixel units arranged in a preset array, each image pixelunit comprises a plurality of original pixels, and each photosensitivepixel corresponds to one original pixel; converting the color-blockimage into a simulation image, wherein, the simulation image comprisessimulation pixels arranged in an array, the simulation pixels comprisecurrent pixels, the original pixels comprise association pixelscorresponding to positions of the current pixels; wherein converting thecolor-block image into the simulation image comprises: determiningwhether a color of a first current pixel is identical to a firstassociation pixel; when the color of the first current pixel isidentical to the first association pixel, determining a pixel value ofthe first association pixel as a pixel value of the first current pixel;and when the color of the first current pixel is different from theassociation pixel, calculating the pixel value of the first currentpixel according to a pixel value of an association pixel unit by a firstinterpolation algorithm, wherein an image pixel unit comprises theassociation pixel unit, the association pixel unit has the same colorwith the first current pixel and is adjacent to the first associationpixel; and converting the simulation image into a simulation true-colorimage.
 2. The method according to claim 1, wherein the preset arraycomprises a Bayer array.
 3. The method according to claim 1, whereineach image pixel unit comprises original pixels arranged in a 2-by-2array.
 4. The method according to claim 1, wherein calculating the pixelvalue of the first current pixel according to the pixel value of theassociation pixel unit by the first interpolation algorithm comprises:calculating a gradual change of the association pixels in eachdirection; calculating a weight of the association pixels in eachdirection; and calculating the pixel value of the first current pixelaccording to the weights and the gradual changes.
 5. The methodaccording to claim 1, further comprising: performing a white-balancecompensation on the color-block image; and performing a reversewhite-balance compensation on the simulation image.
 6. The methodaccording to claim 1, further comprising: performing at least one of abad-point compensation and a crosstalk compensation on the color-blockimage.
 7. The method according to claim 1, further comprising:performing at least one of a mirror shape correction, a demosaickingprocessing, a denoising processing and an edge sharpening processing onthe simulation image.
 8. The image processing method according to claim1, wherein converting the merged image into the merged true-color imagecomprises: converting the merged image into a restoration imagecorresponding to the simulation image using a second interpolationalgorithm, wherein the restoration image comprises restoration pixelsarranged in an array, and a complexity of the second interpolationalgorithm is less than that of the first interpolation algorithm; andconverting the restoration image into the merged true-color image.
 9. Acontrol apparatus, applied for controlling an electronic device, whereinthe electronic device comprises an imaging apparatus and a displayer,the imaging apparatus comprises an image sensor, the image sensorcomprises an array of photosensitive pixel units and an array of filterunits arranged on the array of photosensitive pixel units, each filterunit corresponds to one photosensitive pixel unit, and eachphotosensitive pixel unit comprises a plurality of photosensitivepixels; the control apparatus comprises a non-transitorycomputer-readable medium comprising computer-executable instructionsstored thereon, and an instruction execution system which is configuredby the instructions to implement following acts: controlling the imagesensor to output a merged image, wherein, the merged image comprises anarray of merged pixels, and a plurality of photosensitive pixels in eachphotosensitive pixel unit are collectively output as one merged pixel;dividing the merged image into a plurality of analysis regions arrangedin an array; computing a phase difference of each of the plurality ofanalysis regions; merging analysis regions each with a phase differenceconforming to a preset condition into a focusing area; identifyingwhether there is a face in the focusing area; when there is the face inthe focusing area, converting the merged image into a merged true-colorimage; when there is no face in the focusing area, determining whether abrightness of the focusing area is less than or equal to a brightnessthreshold, whether a green proportion of the focusing area is less thanor equal to a proportion threshold and whether a space frequency of thefocusing area is less than or equal to a frequency threshold; when thebrightness of the focusing area is less than or equal to the brightnessthreshold, the green proportion is less than or equal to the proportionthreshold and the space frequency is less than or equal to the frequencythreshold, converting the merged image into the merged true-color image;when the brightness of the focusing area is greater than the brightnessthreshold, the green proportion is greater than the proportion thresholdand the space frequency is greater than the frequency threshold,controlling the image sensor to output a color-block image, wherein, thecolor-block image comprises image pixel units arranged in a presetarray, each image pixel unit comprises a plurality of original pixels,and each photosensitive pixel corresponds to one original pixel;converting the color-block image into a simulation image, wherein, thesimulation image comprises simulation pixels arranged in an array, thesimulation pixels comprise current pixels, the original pixels compriseassociation pixels corresponding to positions of the current pixels;wherein converting the color-block image into the simulation imagecomprises: determining whether a color of a first current pixel isidentical to a first association pixel; when the color of the firstcurrent pixel is identical to the first association pixel, determining apixel value of the first association pixel as a pixel value of the firstcurrent pixel; and when the color of the first current pixel isdifferent from the first association pixel, calculating the pixel valueof the first current pixel according to a pixel value of an associationpixel unit by a first interpolation algorithm, wherein an image pixelunit comprises the association pixel unit, the association pixel unithas the same color with the first current pixel and is adjacent to thefirst association pixel; and converting the simulation image into asimulation true-color image.
 10. The control apparatus according toclaim 9, wherein the instruction execution system is configured tocalculate the pixel value of the first current pixel according to thepixel value of the association pixel unit by the first interpolationalgorithm based on following acts: calculating a gradual change of theassociation pixels in each direction; calculating a weight of theassociation pixels in each direction; and calculating the pixel value ofthe first current pixel according to the weights and the gradualchanges.
 11. The control apparatus according to claim 9, wherein theinstruction execution system is configured to implement acts of:performing a white-balance compensation on the color-block image; andperforming a reverse white-balance compensation on the simulation image.12. The control apparatus according to claim 9, wherein the instructionexecution system is configured to implement acts of: performing at leastone of a bad-point compensation and a crosstalk compensation on thecolor-block image.
 13. The control apparatus according to claim 9,wherein the instruction execution system is configured to implement actsof: performing at least one of a mirror shape correction, a demosaickingprocessing, a denoising processing and an edge sharpening processing onthe simulation image.
 14. The control apparatus according to claim 9,wherein the instruction execution system is configured to convert themerged image into the merged true-color image by acts of: converting themerged image into a restoration image corresponding to the simulationimage using a second interpolation algorithm, wherein the restorationimage comprises restoration pixels arranged in an array, and acomplexity of the second interpolation algorithm is less than that ofthe first interpolation algorithm; and converting the restoration imageinto the merged true-color image.
 15. An electronic device, comprising:an imaging apparatus, wherein the imaging apparatus comprises an imagesensor, the image sensor comprises an array of photosensitive pixelunits and an array of filter units arranged on the array ofphotosensitive pixel units, each filter unit corresponds to onephotosensitive pixel unit, and each photosensitive pixel unit comprisesa plurality of photosensitive pixels; a displayer; and a controlapparatus, wherein the control apparatus comprises a non-transitorycomputer-readable medium comprising computer-executable instructionsstored thereon, and an instruction execution system which is configuredby the instructions to implement following acts: controlling the imagesensor to output a merged image, wherein, the merged image comprises anarray of merged pixels, and a plurality of photosensitive pixels in eachphotosensitive pixel unit are collectively output as one merged pixel;dividing the merged image into a plurality of analysis regions arrangedin an array; computing a phase difference of each of the plurality ofanalysis regions; merging analysis regions each with a phase differenceconforming to a preset condition into a focusing area; identifyingwhether there is a face in the focusing area; when there is the face inthe focusing area, converting the merged image into a merged true-colorimage; when there is no face in the focusing area, determining whether abrightness of the focusing area is less than or equal to a brightnessthreshold, whether a green proportion of the focusing area is less thanor equal to a proportion threshold and whether a space frequency of thefocusing area is less than or equal to a frequency threshold; when thebrightness of the focusing area is less than or equal to the brightnessthreshold, the green proportion is less than or equal to the proportionthreshold and the space frequency is less than or equal to the frequencythreshold, converting the merged image into the merged true-color image;when the brightness of the focusing area is greater than the brightnessthreshold, the green proportion is greater than the proportion thresholdand the space frequency is greater than the frequency threshold,controlling the image sensor to output a color-block image, wherein, thecolor-block image comprises image pixel units arranged in a presetarray, each image pixel unit comprises a plurality of original pixels,and each photosensitive pixel corresponds to one original pixel;converting the color-block image into a simulation image, wherein, thesimulation image comprises simulation pixels arranged in an array, thesimulation pixels comprise current pixels, the original pixels compriseassociation pixels corresponding to positions of the current pixels;wherein converting the color-block image into the simulation imagecomprises: determining whether a color of a first current pixel isidentical to a first association pixel; when the color of the firstcurrent pixel is identical to that of the first association pixel,determining a pixel value of the first association pixel as a pixelvalue of the first current pixel; and when the color of the firstcurrent pixel is different from the first association pixel, calculatingthe pixel value of the first current pixel according to a pixel value ofan association pixel unit by a first interpolation algorithm, wherein animage pixel unit comprises the association pixel unit, the associationpixel unit has the same color with the first current pixel and isadjacent to the first association pixel; and converting the simulationimage into a simulation true-color image.
 16. The electronic deviceaccording to claim 15, wherein the instruction execution system isconfigured to calculate the pixel value of the first current pixelaccording to the pixel value of the association pixel unit by the firstinterpolation algorithm based on following acts: calculating a gradualchange of the association pixels in each direction; calculating a weightof the association pixels in each direction; and calculating the pixelvalue of the first current pixel according to the weights and thegradual changes.
 17. The electronic device according to claim 15,wherein the instruction execution system is configured to implement actsof: performing a white-balance compensation on the color-block image;and performing a reverse white-balance compensation on the simulationimage.
 18. The electronic device according to claim 15, wherein theinstruction execution system is configured to implement acts of:performing at least one of a bad-point compensation and a crosstalkcompensation on the color-block image.
 19. The electronic deviceaccording to claim 15, wherein the instruction execution system isconfigured to implement acts of: performing at least one of a mirrorshape correction, a demosaicking processing, a denoising processing andan edge sharpening processing on the simulation image.
 20. Theelectronic device according to claim 15, wherein the instructionexecution system is configured to convert the merged image into themerged true-color image by acts of: converting the merged image into arestoration image corresponding to the simulation image using a secondinterpolation algorithm, wherein the restoration image comprisesrestoration pixels arranged in an array, and a complexity of the secondinterpolation algorithm is less than that of the first interpolationalgorithm; and converting the restoration image into the mergedtrue-color image.